{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8888e101",
   "metadata": {},
   "source": [
    "## Elastic Search Re Ranker  \n",
    "This notebooks shows you how you can create a elastic search connection, how can you access Elastic search instance and apply Re-Ranker model on results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f3ee51e",
   "metadata": {},
   "source": [
    "<span style=\"color:blueviolet\">Step 1. Import Pre-required libs</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0ee97809",
   "metadata": {},
   "outputs": [],
   "source": [
    "from elasticsearch import Elasticsearch\n",
    "from elasticsearch.exceptions import RequestError\n",
    "import pandas as pd\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af6a5e8b",
   "metadata": {},
   "source": [
    "<span style=\"color:blueviolet\">Step 2. Skip unwanted chars from document</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "81e4996d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def skip_unwanted_characters(document, keyword):\n",
    "        lines = document.split('\\n')\n",
    "        desired_text = \"\"\n",
    "        last_occurrence = -1\n",
    "        for i, line in enumerate(lines):\n",
    "            if keyword in line:\n",
    "                last_occurrence = i\n",
    "                \n",
    "        if last_occurrence != -1:\n",
    "            for line in lines[last_occurrence+1:]:\n",
    "                desired_text += line.strip() + \"\\n\"\n",
    "        else:\n",
    "            desired_text = document\n",
    "    \n",
    "        return desired_text.strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4418422",
   "metadata": {},
   "source": [
    "<span style=\"color:blueviolet\">Step 3. Removing HTML tags from document</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e26d962c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def pre_processingtext(text_data):\n",
    "        replaced = re.sub(\"\\{{ .*?\\}}\", \"\", text_data)\n",
    "        replaced = re.sub(\"\\{: .*?\\}\", \"\", text_data)\n",
    "        replaced = re.sub(\"\\(.*?\\)|\\[.*?\\] |\\{.*?\\}\", \"\", replaced)\n",
    "        replaced = re.sub(\"</?div[^>]*>\", \"\", replaced)\n",
    "        replaced = re.sub(\"</?p[^>]*>\", \"\", replaced)\n",
    "        replaced = re.sub(\"</?a[^>]*>\", \"\", replaced)\n",
    "        replaced = re.sub(\"</?h*[^>]*>\", \"\", replaced)\n",
    "        replaced = re.sub(\"</?em*[^>]*>\", \"\", replaced)\n",
    "        replaced = re.sub(\"</?img*[^>]*>\", \"\", replaced)\n",
    "        replaced = re.sub(\"&amp;\", \"\", replaced)\n",
    "        replaced = re.sub(\"</?href*>\", \"\", replaced)\n",
    "        replaced = replaced.replace(\"}\",\"\")\n",
    "        replaced = replaced.replace(\"##\",\"\")\n",
    "        replaced = replaced.replace(\"###\",\"\")\n",
    "        replaced = replaced.replace(\"#\",\"\")\n",
    "        replaced = replaced.replace(\"*\",\"\")\n",
    "        replaced = replaced.replace(\"<strong>\",\"\")\n",
    "        replaced = replaced.replace(\"</strong>\",\"\")\n",
    "        replaced = replaced.replace(\"<ul>\",\"\")\n",
    "        replaced = replaced.replace(\"</ul>\",\"\")\n",
    "        replaced = replaced.replace(\"<li>\",\"\")\n",
    "        replaced = replaced.replace(\"</li>\",\"\")\n",
    "        replaced = replaced.replace(\"<ol>\",\"\")\n",
    "        replaced = replaced.replace(\"</ol>\",\"\")\n",
    "        return replaced"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86ce8c0f",
   "metadata": {},
   "source": [
    "<span style=\"color:blueviolet\">Step 4. By using below method we can get documnets from elastic search</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "ccb09587",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_retervier(question):\n",
    "    results_list=[]\n",
    "    # Create an instance of Elasticsearch with TLS options\n",
    "    es_client = Elasticsearch(\n",
    "    'https://ibm_cloud_fb86f339_5b26_42c8_b4d6_b9663ffa4fdd:3e6a2eb311351ad1451364cb2713e531eea8e4797af044e085bc1bb52e8fc0f9@3d862675-f715-499c-b9e4-ffba4d8321a0.2adb0220806343e3ae11df79c89b377f.databases.appdomain.cloud:32062',\n",
    "    ca_certs='/Users/abhilashamangal/Downloads/5cb6eb86-ae1c-11e9-99c9-6a007ab2fc0b'\n",
    "    )\n",
    "    info = es_client.info()\n",
    "    print(info)\n",
    "    index_name = 'superknowa'\n",
    "    search_query5 ={\n",
    "      \"query\": {\n",
    "        \"bool\": {\n",
    "          \"must\": [\n",
    "        { \"match\": { \"content\": \"'+question+'\" }}\n",
    "      ],\n",
    "          \"must_not\": [\n",
    "        { \"match\": { \"published_source\": \"Redbooks\" }}\n",
    "      ]\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "    ## Top 10 documents \n",
    "    response = es_client.search(\n",
    "    index=index_name,\n",
    "    body=search_query5,\n",
    "    scroll='5m',  # Set the scroll timeout (e.g., 5 minutes)\n",
    "    size=10  # Set the number of documents to retrieve per scroll\n",
    "    )\n",
    "    all_hits = response['hits']['hits']\n",
    "    i =0\n",
    "    print(len(all_hits))\n",
    "    for num, doc in enumerate(all_hits):\n",
    "        doc_id = doc[\"_source\"][\"id\"]\n",
    "        doc_url = doc[\"_source\"][\"url\"].replace(\" \",\"\")\n",
    "        doc_source = doc[\"_source\"][\"published_source\"]\n",
    "        print (\"DOC URL:\", doc[\"_source\"][\"url\"], \"--->\", \"\\n\")\n",
    "        print (\"DOC id:\", doc[\"_source\"][\"id\"], \"--->\", \"\\n\")\n",
    "        print (\"DOC Source:\", doc[\"_source\"][\"published_source\"], \"\\n\")\n",
    "        #print (\"DOC content:\", doc[\"_source\"][\"content\"], \"\\n\")\n",
    "        string_unicode = doc[\"_source\"][\"content\"]\n",
    "        doc = string_unicode.encode(\"ascii\", \"ignore\")\n",
    "        string_decode = doc.decode()\n",
    "        keyword = \"{: shortdesc} \"\n",
    "        cleaned_text = skip_unwanted_characters(string_decode,keyword)\n",
    "        pattern =  r'\\{\\s*:\\s*[\\w#-]+\\s*\\}|\\{\\s*:\\s*\\w+\\s*\\}|\\n\\s*\\n'\n",
    "        cleaned_text = re.sub(pattern, '', cleaned_text)\n",
    "        cleaned_text = pre_processingtext(cleaned_text)\n",
    "        query_hits = {\n",
    "                    \"document\": {\n",
    "                        \"rank\": i,\n",
    "                        \"document_id\": doc_id,\n",
    "                        \"text\": cleaned_text[0:4000], \n",
    "                        \"url\" : doc_url,\n",
    "                        \"source\":doc_source\n",
    "                    },\n",
    "                }\n",
    "\n",
    "        results_list.append(query_hits)\n",
    "        results_to_display = [results_list['document'] for results_list in results_list]\n",
    "        df = pd.DataFrame.from_records(results_to_display, columns=['rank','document_id','text','url','source'])\n",
    "        # df['title'] = np.random.randint(1, 10, df.shape[0])\n",
    "        df.dropna(inplace=True)\n",
    "        i = i+1\n",
    "    return results_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "b7326953",
   "metadata": {},
   "outputs": [],
   "source": [
    "question=\"A 2-node IBM Sterling B2B Integrator (SBI) clustered solution is being designed. The customer will initiate connections with external partners using HTTP, FTP and SFTP protocols. For load balancing and fail over purposes 2 client adapters for each protocol are configured on each node and Service Groups are being used. How many Service Groups\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "59e4754d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'm-1.3d862675-f715-499c-b9e4-ffba4d8321a0.d5c42fad68fd498ba08f6af6107b71cd.2adb0220806343e3ae11df79c89b377f.databases.appdomain.cloud', 'cluster_name': '3d862675-f715-499c-b9e4-ffba4d8321a0', 'cluster_uuid': 'UsM9ak-LRYajVAwt5yeQxw', 'version': {'number': '7.10.2', 'build_flavor': 'oss', 'build_type': 'tar', 'build_hash': '747e1cc71def077253878a59143c1f785afa92b9', 'build_date': '2021-01-13T00:42:12.435326Z', 'build_snapshot': False, 'lucene_version': '8.7.0', 'minimum_wire_compatibility_version': '6.8.0', 'minimum_index_compatibility_version': '6.0.0-beta1'}, 'tagline': 'You Know, for Search'}\n",
      "10\n",
      "DOC URL: \"https://github.com/ibm-cloud-docs/understand-questions.md\" ---> \n",
      "\n",
      "DOC id: watson-assistant_understand-questions ---> \n",
      "\n",
      "DOC Source: IBM Developer docs \n",
      "\n",
      "DOC URL: \"https://github.com/ibm-cloud-docs/change-topic.md\" ---> \n",
      "\n",
      "DOC id: watson-assistant_change-topic ---> \n",
      "\n",
      "DOC Source: IBM Developer docs \n",
      "\n",
      "DOC URL: https://medium.com/ibm-watson/three-ways-to-improve-your-conversational-ai-a33db7fc0d5?source=collection_archive---------0----------------------- ---> \n",
      "\n",
      "DOC id: Three ways to improve your conversational AI ---> \n",
      "\n",
      "DOC Source: Medium \n",
      "\n",
      "DOC URL: https://medium.com/ibm-watson/three-ways-to-improve-your-conversational-ai-a33db7fc0d5?source=collection_archive---------0----------------------- ---> \n",
      "\n",
      "DOC id: Three ways to improve your conversational AI ---> \n",
      "\n",
      "DOC Source: Medium \n",
      "\n",
      "DOC URL: https://medium.com/ibm-watson/three-ways-to-improve-your-conversational-ai-a33db7fc0d5?source=collection_archive---------0----------------------- ---> \n",
      "\n",
      "DOC id: Three ways to improve your conversational AI ---> \n",
      "\n",
      "DOC Source: Medium \n",
      "\n",
      "DOC URL: https://medium.com/ibm-watson/three-ways-to-improve-your-conversational-ai-a33db7fc0d5?source=collection_archive---------0----------------------- ---> \n",
      "\n",
      "DOC id: Three ways to improve your conversational AI ---> \n",
      "\n",
      "DOC Source: Medium \n",
      "\n",
      "DOC URL: https://developer.ibm.com/blogs/ /tutorials/create-your-first-assistant-powered-chatbot\n",
      " ---> \n",
      "\n",
      "DOC id:  Create a retail customer service chatbot\n",
      " ---> \n",
      "\n",
      "DOC Source: IBM Developer \n",
      "\n",
      "DOC URL: https://developer.ibm.com/blogs/ /articles/develop-an-ai-infused-automation-tool-to-convert-business-faqs-to-watson-assistant-ready-chatbot-input\n",
      " ---> \n",
      "\n",
      "DOC id:  Archived | Develop an AI-infused automation tool to convert business FAQs to Watson Assistant-ready chatbot input\n",
      " ---> \n",
      "\n",
      "DOC Source: IBM Developer \n",
      "\n",
      "DOC URL: https://medium.com/ibm-data-ai/finding-concise-answers-to-questions-in-enterprise-documents-53a865898dbd?source=collection_archive---------2----------------------- ---> \n",
      "\n",
      "DOC id: Finding concise answers to questions in enterprise documents ---> \n",
      "\n",
      "DOC Source: Medium \n",
      "\n",
      "DOC URL: https://medium.com/ibm-data-ai/finding-concise-answers-to-questions-in-enterprise-documents-53a865898dbd?source=collection_archive---------2----------------------- ---> \n",
      "\n",
      "DOC id: Finding concise answers to questions in enterprise documents ---> \n",
      "\n",
      "DOC Source: Medium \n",
      "\n"
     ]
    }
   ],
   "source": [
    "results_list =elastic_retervier(question)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4abf4149",
   "metadata": {},
   "source": [
    "### 2. ColBERT Re-Ranker model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "a4ad9340",
   "metadata": {},
   "outputs": [],
   "source": [
    "from primeqa.components.reranker.colbert_reranker import ColBERTReranker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "988da1e7",
   "metadata": {},
   "source": [
    "<span style=\"color:blueviolet\">Step 5. By using below code you can download the model</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "bceee9b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-07-17 18:24:17--  https://huggingface.co/PrimeQA/DrDecr_XOR-TyDi_whitebox/resolve/main/DrDecr.dnn\n",
      "Resolving huggingface.co (huggingface.co)... 13.35.191.115, 13.35.191.48, 13.35.191.58, ...\n",
      "Connecting to huggingface.co (huggingface.co)|13.35.191.115|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://cdn-lfs.huggingface.co/repos/d4/ef/d4ef44ce7d987b0ad737d45af61c195b32745b69da94de28f652bef09436ef7d/b9243c4014ae3fc2d779c6560900962d26262ec76137f76140c9f95154ca9522?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27DrDecr.dnn%3B+filename%3D%22DrDecr.dnn%22%3B&Expires=1689857658&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTY4OTg1NzY1OH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy5odWdnaW5nZmFjZS5jby9yZXBvcy9kNC9lZi9kNGVmNDRjZTdkOTg3YjBhZDczN2Q0NWFmNjFjMTk1YjMyNzQ1YjY5ZGE5NGRlMjhmNjUyYmVmMDk0MzZlZjdkL2I5MjQzYzQwMTRhZTNmYzJkNzc5YzY1NjA5MDA5NjJkMjYyNjJlYzc2MTM3Zjc2MTQwYzlmOTUxNTRjYTk1MjI%7EcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj0qIn1dfQ__&Signature=BZOLV5dQoeFfK50HCbauTdSF%7EttLeUEgrVyQ4hFGUGWTwNe1-3psMiWR9%7E4nxPaTV8zbIILls9LKx7RFeVKHq%7Euk5xLniqvQ8-uRLgtICGYqpDO9y7P77wri9V86nAlOfmVGr2b8YhBgzH2l9CUfKjG6GbiS%7EPDynVqd5wgKvD9Cz127dwHUATZ3g6iJBW5QYuD1NkkfdnWNL%7EQz4JsqdPW06TB2aucelqBngrJj%7E7WSPI20ZGEcYbro1Uc4oXd0nsrr-jD9e22a-C1Fv2S9NPIv2pSpyan2Wz8hXgQYKMny9bGaCsAosJG1pp%7EI5yvz12bb8rvNWXnBX36-sVLbyw__&Key-Pair-Id=KVTP0A1DKRTAX [following]\n",
      "--2023-07-17 18:24:17--  https://cdn-lfs.huggingface.co/repos/d4/ef/d4ef44ce7d987b0ad737d45af61c195b32745b69da94de28f652bef09436ef7d/b9243c4014ae3fc2d779c6560900962d26262ec76137f76140c9f95154ca9522?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27DrDecr.dnn%3B+filename%3D%22DrDecr.dnn%22%3B&Expires=1689857658&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTY4OTg1NzY1OH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy5odWdnaW5nZmFjZS5jby9yZXBvcy9kNC9lZi9kNGVmNDRjZTdkOTg3YjBhZDczN2Q0NWFmNjFjMTk1YjMyNzQ1YjY5ZGE5NGRlMjhmNjUyYmVmMDk0MzZlZjdkL2I5MjQzYzQwMTRhZTNmYzJkNzc5YzY1NjA5MDA5NjJkMjYyNjJlYzc2MTM3Zjc2MTQwYzlmOTUxNTRjYTk1MjI%7EcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj0qIn1dfQ__&Signature=BZOLV5dQoeFfK50HCbauTdSF%7EttLeUEgrVyQ4hFGUGWTwNe1-3psMiWR9%7E4nxPaTV8zbIILls9LKx7RFeVKHq%7Euk5xLniqvQ8-uRLgtICGYqpDO9y7P77wri9V86nAlOfmVGr2b8YhBgzH2l9CUfKjG6GbiS%7EPDynVqd5wgKvD9Cz127dwHUATZ3g6iJBW5QYuD1NkkfdnWNL%7EQz4JsqdPW06TB2aucelqBngrJj%7E7WSPI20ZGEcYbro1Uc4oXd0nsrr-jD9e22a-C1Fv2S9NPIv2pSpyan2Wz8hXgQYKMny9bGaCsAosJG1pp%7EI5yvz12bb8rvNWXnBX36-sVLbyw__&Key-Pair-Id=KVTP0A1DKRTAX\n",
      "Resolving cdn-lfs.huggingface.co (cdn-lfs.huggingface.co)... 18.164.202.109, 18.164.202.9, 18.164.202.55, ...\n",
      "Connecting to cdn-lfs.huggingface.co (cdn-lfs.huggingface.co)|18.164.202.109|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 3333244115 (3.1G) [binary/octet-stream]\n",
      "Saving to: ‘DrDecr.dnn.2’\n",
      "\n",
      "DrDecr.dnn.2        100%[===================>]   3.10G  14.4MB/s    in 3m 43s  \n",
      "\n",
      "2023-07-17 18:28:00 (14.3 MB/s) - ‘DrDecr.dnn.2’ saved [3333244115/3333244115]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#!wget https://huggingface.co/PrimeQA/DrDecr_XOR-TyDi_whitebox/resolve/main/DrDecr.dnn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5da4ef2",
   "metadata": {},
   "source": [
    "<span style=\"color:blueviolet\">Step 6. Please provide model path to load the model</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "b14811b1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Jul 17, 18:28:04] #>>>>> at ColBERT name (model type) : DrDecr.dnn\n",
      "[Jul 17, 18:28:04] #>>>>> at BaseColBERT name (model type) : DrDecr.dnn\n",
      "[Jul 17, 18:28:07] factory model type: xlm-roberta-base\n",
      "[Jul 17, 18:28:18] get query model type: xlm-roberta-base\n",
      "[Jul 17, 18:28:22] get doc model type: xlm-roberta-base\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/torch/cuda/amp/grad_scaler.py:115: UserWarning: torch.cuda.amp.GradScaler is enabled, but CUDA is not available.  Disabling.\n",
      "  warnings.warn(\"torch.cuda.amp.GradScaler is enabled, but CUDA is not available.  Disabling.\")\n"
     ]
    }
   ],
   "source": [
    "# model_name_or_path=\"ibm/re2g-reranker-nq\"\n",
    "reranker = ColBERTReranker(model=\"DrDecr.dnn\")\n",
    "reranker.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a70c2f63",
   "metadata": {},
   "source": [
    "<span style=\"color:blueviolet\">Step 7. By using below method apply re-ranked algo on documents</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "bd6f2e9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_reranker(question, max_reranked_documents, reranker):\n",
    "        results_list = elastic_retervier(question)\n",
    "        if len(results_list) >0:\n",
    "            reranked_results = reranker.predict(queries= [question], documents = [results_list], max_num_documents=max_reranked_documents)\n",
    "\n",
    "            #print(reranked_results)\n",
    "\n",
    "            reranked_results_to_display = [result['document'] for result in reranked_results[0]]\n",
    "            df = pd.DataFrame.from_records(reranked_results_to_display, columns=['rank','document_id','text','url','source'])\n",
    "            #print('======================================================================')\n",
    "            #print(f'QUERY: {question}')\n",
    "            #display( HTML(df.to_html()) )\n",
    "            return df['text'][0] , df['url'][0]\n",
    "        else:\n",
    "            return \"0 documents found\" , \"None\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "8aad9fd7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'm-2.3d862675-f715-499c-b9e4-ffba4d8321a0.d5c42fad68fd498ba08f6af6107b71cd.2adb0220806343e3ae11df79c89b377f.databases.appdomain.cloud', 'cluster_name': '3d862675-f715-499c-b9e4-ffba4d8321a0', 'cluster_uuid': 'UsM9ak-LRYajVAwt5yeQxw', 'version': {'number': '7.10.2', 'build_flavor': 'oss', 'build_type': 'tar', 'build_hash': '747e1cc71def077253878a59143c1f785afa92b9', 'build_date': '2021-01-13T00:42:12.435326Z', 'build_snapshot': False, 'lucene_version': '8.7.0', 'minimum_wire_compatibility_version': '6.8.0', 'minimum_index_compatibility_version': '6.0.0-beta1'}, 'tagline': 'You Know, for Search'}\n",
      "10\n",
      "DOC URL: \"https://github.com/ibm-cloud-docs/understand-questions.md\" ---> \n",
      "\n",
      "DOC id: watson-assistant_understand-questions ---> \n",
      "\n",
      "DOC Source: IBM Developer docs \n",
      "\n",
      "DOC URL: \"https://github.com/ibm-cloud-docs/change-topic.md\" ---> \n",
      "\n",
      "DOC id: watson-assistant_change-topic ---> \n",
      "\n",
      "DOC Source: IBM Developer docs \n",
      "\n",
      "DOC URL: https://medium.com/ibm-watson/three-ways-to-improve-your-conversational-ai-a33db7fc0d5?source=collection_archive---------0----------------------- ---> \n",
      "\n",
      "DOC id: Three ways to improve your conversational AI ---> \n",
      "\n",
      "DOC Source: Medium \n",
      "\n",
      "DOC URL: https://medium.com/ibm-watson/three-ways-to-improve-your-conversational-ai-a33db7fc0d5?source=collection_archive---------0----------------------- ---> \n",
      "\n",
      "DOC id: Three ways to improve your conversational AI ---> \n",
      "\n",
      "DOC Source: Medium \n",
      "\n",
      "DOC URL: https://medium.com/ibm-watson/three-ways-to-improve-your-conversational-ai-a33db7fc0d5?source=collection_archive---------0----------------------- ---> \n",
      "\n",
      "DOC id: Three ways to improve your conversational AI ---> \n",
      "\n",
      "DOC Source: Medium \n",
      "\n",
      "DOC URL: https://medium.com/ibm-watson/three-ways-to-improve-your-conversational-ai-a33db7fc0d5?source=collection_archive---------0----------------------- ---> \n",
      "\n",
      "DOC id: Three ways to improve your conversational AI ---> \n",
      "\n",
      "DOC Source: Medium \n",
      "\n",
      "DOC URL: https://developer.ibm.com/blogs/ /tutorials/create-your-first-assistant-powered-chatbot\n",
      " ---> \n",
      "\n",
      "DOC id:  Create a retail customer service chatbot\n",
      " ---> \n",
      "\n",
      "DOC Source: IBM Developer \n",
      "\n",
      "DOC URL: https://developer.ibm.com/blogs/ /articles/develop-an-ai-infused-automation-tool-to-convert-business-faqs-to-watson-assistant-ready-chatbot-input\n",
      " ---> \n",
      "\n",
      "DOC id:  Archived | Develop an AI-infused automation tool to convert business FAQs to Watson Assistant-ready chatbot input\n",
      " ---> \n",
      "\n",
      "DOC Source: IBM Developer \n",
      "\n",
      "DOC URL: https://medium.com/ibm-data-ai/finding-concise-answers-to-questions-in-enterprise-documents-53a865898dbd?source=collection_archive---------2----------------------- ---> \n",
      "\n",
      "DOC id: Finding concise answers to questions in enterprise documents ---> \n",
      "\n",
      "DOC Source: Medium \n",
      "\n",
      "DOC URL: https://medium.com/ibm-data-ai/finding-concise-answers-to-questions-in-enterprise-documents-53a865898dbd?source=collection_archive---------2----------------------- ---> \n",
      "\n",
      "DOC id: Finding concise answers to questions in enterprise documents ---> \n",
      "\n",
      "DOC Source: Medium \n",
      "\n",
      "[Jul 17, 18:28:27] #> XMLR QueryTokenizer.tensorize(batch_text[0], batch_background[0], bsize) ==\n",
      "[Jul 17, 18:28:27] #> Input: $ A 2-node IBM Sterling B2B Integrator (SBI) clustered solution is being designed. The customer will initiate connections with external partners using HTTP, FTP and SFTP protocols. For load balancing and fail over purposes 2 client adapters for each protocol are configured on each node and Service Groups are being used. How many Service Groups, \t\t True, \t\t None\n",
      "[Jul 17, 18:28:27] #> Output IDs: torch.Size([32]), tensor([     0,   9748,     62,   4720,    157,    112,  90540,  94358,   2069,\n",
      "           335,    304,    571,  91969,   1290,     15,    294,  16312,     16,\n",
      "        234737,    297,  29806,     83,   8035,  82775,      5,    581,  43373,\n",
      "          1221, 173969,     13,  94878,      2])\n",
      "[Jul 17, 18:28:27] #> Output Mask: torch.Size([32]), tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 1, 1, 1, 1, 1, 1, 1])\n",
      "[Jul 17, 18:28:27] #>>>> colbert query ==\n",
      "[Jul 17, 18:28:27] #>>>>> input_ids: torch.Size([32]), tensor([     0,   9748,     62,   4720,    157,    112,  90540,  94358,   2069,\n",
      "           335,    304,    571,  91969,   1290,     15,    294,  16312,     16,\n",
      "        234737,    297,  29806,     83,   8035,  82775,      5,    581,  43373,\n",
      "          1221, 173969,     13,  94878,      2])\n",
      "[Jul 17, 18:28:28] #>>>> before linear query ==\n",
      "[Jul 17, 18:28:28] #>>>>> Q: torch.Size([32, 768]), tensor([[ 0.0464,  0.2072,  0.2129,  ..., -0.1865,  0.0969,  0.3335],\n",
      "        [-0.0566, -0.3528,  0.0886,  ..., -0.3468,  0.2169,  0.9352],\n",
      "        [ 0.1047, -0.2178,  0.3751,  ..., -0.1486,  0.2769,  0.9438],\n",
      "        ...,\n",
      "        [ 0.1704, -0.0844,  0.0938,  ...,  0.5308,  0.0214,  0.8489],\n",
      "        [ 0.1666,  0.3379, -0.0154,  ...,  0.5230, -0.0272,  1.2076],\n",
      "        [-0.2166, -0.2147,  0.4273,  ...,  0.0955,  0.1771,  0.9126]])\n",
      "[Jul 17, 18:28:28] #>>>>> self.linear query : Parameter containing:\n",
      "tensor([[-0.0286,  0.0017, -0.0202,  ..., -0.0262,  0.0210,  0.0006],\n",
      "        [-0.0102,  0.0121, -0.0111,  ..., -0.0362, -0.0165, -0.0012],\n",
      "        [-0.0047, -0.0172, -0.0054,  ..., -0.0069, -0.0194, -0.0193],\n",
      "        ...,\n",
      "        [-0.0286,  0.0231,  0.0004,  ...,  0.0373, -0.0045,  0.0125],\n",
      "        [ 0.0051,  0.0023,  0.0212,  ..., -0.0254,  0.0034,  0.0206],\n",
      "        [-0.0068,  0.0256, -0.0263,  ...,  0.0200,  0.0125, -0.0149]],\n",
      "       requires_grad=True)\n",
      "[Jul 17, 18:28:28] #>>>> colbert query ==\n",
      "[Jul 17, 18:28:28] #>>>>> Q: torch.Size([32, 128]), tensor([[ 2.0222e-01,  1.5244e-01,  3.4942e-01,  ...,  2.3762e-01,\n",
      "          3.0797e-01, -1.8854e-01],\n",
      "        [ 8.5492e-01,  2.6937e-01,  6.7730e-01,  ...,  2.6736e-01,\n",
      "          2.1526e-01, -3.4628e-01],\n",
      "        [ 7.8804e-01,  8.9284e-02,  1.0728e+00,  ...,  4.4751e-01,\n",
      "          3.9249e-01, -4.5611e-01],\n",
      "        ...,\n",
      "        [ 6.1470e-01,  3.1656e-01,  4.3243e-01,  ...,  4.6121e-01,\n",
      "          1.5321e-01, -3.3959e-01],\n",
      "        [ 8.9573e-01,  1.0104e-03,  3.8239e-01,  ..., -1.8232e-01,\n",
      "          5.3915e-01, -1.4966e-01],\n",
      "        [ 6.0033e-01,  2.4665e-01,  1.1160e+00,  ...,  8.5761e-01,\n",
      "          4.9804e-01, -2.7413e-01]])\n",
      "[Jul 17, 18:28:28] #> XLMR DocTokenizer.tensorize(batch_text[0], batch_background[0], bsize) ==\n",
      "[Jul 17, 18:28:28] #> Input: $ \n",
      "Understanding your users' questions or requestsActions represent the tasks or questions that your assistant can help customers with. Each action has a beginning and an end, making up a conversation between the assistant and a customer. Learn how to begin an action, where it understands and recognizes a goal based on the words a customer uses to ask a question or make a request.\n",
      "Beginning an actionEach assistant can include as many actions as you need to have conversations with your users. You design each individual action to recognize a specific question or request, and when it does, the action starts.\n",
      "When you create a new action, your first task is to enter one phrase that a customer types or says to start the conversation about a specific topic. This phrase determines the problem that your customer has or the question your user asks.\n",
      "To get going, you need to enter only one phrase, for example: What are your store hours?.\n",
      "After you enter the phrase, it is stored in Customer starts with, at the start of the action.Testing your phraseBefore even doing anything else with your action, you can already start checking that your assistant recognizes the starting phrase.Click the Preview button.\n",
      "Enter your first phrase, for example: What are your store hours?.If you see There are no additional steps for this action, that means the action recognizes the phrase. If the assistant doesn't understand the phrase, you'll see the built-in action No action matches. For more information, see When the assistant can't understand your customer's request.Adding more examplesWhen you're creating a new action, one example phrase is enough to start with. You can build the rest of your action with steps before adding more example phrases. Then, return to Customer starts with and add 10 or more variations of the same question or request, using words that your customers commonly use. For example:Are you open on the weekend?\n",
      "How late are you open today?\n",
      "Get store hours\n",
      "What time do you open?\n",
      "Are you open now?Each phrase can be up to 1,024 characters in length.\n",
      "By adding these phrases, your assistant learns what is the right action for what a customer wants. The additional examples build the training data that the machine learning engine of Watson Assistant uses to create a natural language processing model. The model is customized to understand your uniquely defined actions.\n",
      "If you have many example phrases, you can upload them from a comma-separated value  file than to define them one by one. If you are migrating intent information from the classic  experience to example phrases in the new  experience, see Migrating intents and entities.Collect the phrases into a CSV file. Save the CSV file with UTF-8 encoding and no byte order mark .If you are creating a new CSV file to upload phrases, the format for each line in the file is as follows:\n",
      "    &lt;phrase&gt;\n",
      "    Where &lt;phrase&gt; is the text of a user example phrase. If youre using a spreadsheet to create a CSV file, put all your phrases into column 1, as shown in the following example:If you downloaded intents from the classic experience, the format for each line in the file is as follows:\n",
      "    &lt;phrase&gt;,&lt;intent&gt;\n",
      "    Where &lt;phrase&gt; is the text of a user example phrase, and &lt;intent&gt; is the name of the intent. For example:\n",
      "    Tell me the current weather conditions.,weather_conditions\n",
      "    Is it raining?,weather_conditions\n",
      "    What's the temperature?,weather_conditions\n",
      "Only one intent can be uploaded per action, so the &lt;intent&gt; information listed in the second column of the CSV file must be the same.Go to Customer starts with at the start of the action.Click the Upload icon .Select a file from your computer.\n",
      "The file is validated and uploaded, and the system trains itself on the new data.\n",
      "You can download your example phrases to a CSV file, so you can then upload and reuse them in another  application.Go to Customer starts with at the start of the action.Click the Download i, \t\t None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Jul 17, 18:28:28] #> Output IDs: torch.Size([180]), tensor([     0,   9749,   9626, 144057,    935,  72095,     25,  17582,    707,\n",
      "         50336,      7,  39450,   5256,  33636,     70,  66211,      7,    707,\n",
      "         17582,    450,    935, 195644,    831,   4358,  59463,    678,      5,\n",
      "         98423,  22631,   1556,     10,  86595,    136,    142,   3564,      4,\n",
      "         20662,   1257,     10,  77104,  17721,     70, 195644,    136,     10,\n",
      "         43373,      5, 134031,   3642,     47,   9842,    142,  22631,      4,\n",
      "          7440,    442,  28219,      7,    136, 125296,      7,     10,  69236,\n",
      "         35509,     98,     70,  34153,     10,  43373,   4527,      7,     47,\n",
      "         26458,     10,   9655,    707,   3249,     10,  50336,      5,  67053,\n",
      "           214,    142,  22631,    647,    934, 195644,    831,  26698,    237,\n",
      "          5941,  61972,    237,    398,   3871,     47,    765,  77104,      7,\n",
      "           678,    935,  72095,      5,   2583,   4331,  12638,  11651,  22631,\n",
      "            47, 125296,     10,  29458,   9655,    707,  50336,      4,    136,\n",
      "          3229,    442,  14602,      4,     70,  22631,   4034,      7,      5,\n",
      "         14847,    398,  28282,     10,   3525,  22631,      4,    935,   5117,\n",
      "         66211,     83,     47,  30957,   1632, 113860,    450,     10,  43373,\n",
      "         52895,    707,  17378,     47,   4034,     70,  77104,   1672,     10,\n",
      "         29458,  28451,      5,   3293, 113860,  83324,      7,     70,   2967,\n",
      "           450,    935,  43373,   1556,    707,     70,   9655,    935,  38937,\n",
      "         26458,      7,      5,    717,   2046,   7730,      4,    398,      2])\n",
      "[Jul 17, 18:28:28] #> Output Mask: torch.Size([180]), tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])\n",
      "[Jul 17, 18:28:28] #>>>> colbert doc ==\n",
      "[Jul 17, 18:28:28] #>>>>> input_ids: torch.Size([180]), tensor([     0,   9749,   9626, 144057,    935,  72095,     25,  17582,    707,\n",
      "         50336,      7,  39450,   5256,  33636,     70,  66211,      7,    707,\n",
      "         17582,    450,    935, 195644,    831,   4358,  59463,    678,      5,\n",
      "         98423,  22631,   1556,     10,  86595,    136,    142,   3564,      4,\n",
      "         20662,   1257,     10,  77104,  17721,     70, 195644,    136,     10,\n",
      "         43373,      5, 134031,   3642,     47,   9842,    142,  22631,      4,\n",
      "          7440,    442,  28219,      7,    136, 125296,      7,     10,  69236,\n",
      "         35509,     98,     70,  34153,     10,  43373,   4527,      7,     47,\n",
      "         26458,     10,   9655,    707,   3249,     10,  50336,      5,  67053,\n",
      "           214,    142,  22631,    647,    934, 195644,    831,  26698,    237,\n",
      "          5941,  61972,    237,    398,   3871,     47,    765,  77104,      7,\n",
      "           678,    935,  72095,      5,   2583,   4331,  12638,  11651,  22631,\n",
      "            47, 125296,     10,  29458,   9655,    707,  50336,      4,    136,\n",
      "          3229,    442,  14602,      4,     70,  22631,   4034,      7,      5,\n",
      "         14847,    398,  28282,     10,   3525,  22631,      4,    935,   5117,\n",
      "         66211,     83,     47,  30957,   1632, 113860,    450,     10,  43373,\n",
      "         52895,    707,  17378,     47,   4034,     70,  77104,   1672,     10,\n",
      "         29458,  28451,      5,   3293, 113860,  83324,      7,     70,   2967,\n",
      "           450,    935,  43373,   1556,    707,     70,   9655,    935,  38937,\n",
      "         26458,      7,      5,    717,   2046,   7730,      4,    398,      2])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/torch/autocast_mode.py:162: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling\n",
      "  warnings.warn('User provided device_type of \\'cuda\\', but CUDA is not available. Disabling')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Jul 17, 18:28:31] #>>>> before linear doc ==\n",
      "[Jul 17, 18:28:31] #>>>>> D: torch.Size([180, 768]), tensor([[ 3.5794e-02,  2.4885e-01,  2.7271e-01,  ...,  7.9367e-03,\n",
      "          1.7441e-01,  3.1807e-01],\n",
      "        [-5.7377e-01,  3.0731e-01,  9.1956e-04,  ..., -5.7313e-01,\n",
      "          1.3373e-01,  1.5070e-01],\n",
      "        [-5.1033e-01,  3.9763e-02, -7.9090e-02,  ..., -1.0753e+00,\n",
      "          3.2651e-01,  2.2172e-01],\n",
      "        ...,\n",
      "        [-3.4739e-01,  1.6587e-01, -2.3984e-03,  ...,  1.4348e-02,\n",
      "          3.1133e-01,  5.8044e-01],\n",
      "        [-2.8136e-01,  3.3100e-02, -5.2250e-02,  ..., -1.9857e-02,\n",
      "          8.8519e-02,  6.1383e-01],\n",
      "        [ 3.5644e-02,  2.4873e-01,  2.6242e-01,  ..., -5.7343e-03,\n",
      "          1.5899e-01,  3.3249e-01]])\n",
      "[Jul 17, 18:28:31] #>>>>> self.linear doc : Parameter containing:\n",
      "tensor([[-0.0286,  0.0017, -0.0202,  ..., -0.0262,  0.0210,  0.0006],\n",
      "        [-0.0102,  0.0121, -0.0111,  ..., -0.0362, -0.0165, -0.0012],\n",
      "        [-0.0047, -0.0172, -0.0054,  ..., -0.0069, -0.0194, -0.0193],\n",
      "        ...,\n",
      "        [-0.0286,  0.0231,  0.0004,  ...,  0.0373, -0.0045,  0.0125],\n",
      "        [ 0.0051,  0.0023,  0.0212,  ..., -0.0254,  0.0034,  0.0206],\n",
      "        [-0.0068,  0.0256, -0.0263,  ...,  0.0200,  0.0125, -0.0149]],\n",
      "       requires_grad=True)\n",
      "[Jul 17, 18:28:31] #>>>> colbert doc ==\n",
      "[Jul 17, 18:28:31] #>>>>> D: torch.Size([180, 128]), tensor([[-0.0742,  0.0381,  0.1713,  ...,  0.4138,  0.1915, -0.1133],\n",
      "        [ 0.0300,  0.5513,  0.3454,  ...,  0.2619,  0.3202, -0.3288],\n",
      "        [ 0.4010,  0.9759,  0.6976,  ...,  0.2230,  0.2293, -0.2318],\n",
      "        ...,\n",
      "        [-0.0361, -0.0669,  0.3754,  ...,  1.2051,  0.1311, -0.5536],\n",
      "        [-0.3042,  0.2557,  0.3219,  ...,  0.8911, -0.1299, -0.2265],\n",
      "        [-0.0655,  0.0408,  0.1745,  ...,  0.4153,  0.2003, -0.1138]])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(\"Archived contentArchive date: 2022-11-08This content is no longer being updated or maintained. The content is provided as is. Given the rapid evolution of technology, some content, steps, or illustrations may have changed.Developing an automated solution for creating intents, utterances, and entities directly from a business's frequently asked questions  can help developers and software teams save considerable time because they do not have to write it manually. In this article, learn about an artificial intelligence -infused automation tool to convert a business's FAQs to Watson Assistant-ready chatbot input. You can then easily integrate the solution with minimal effort to various input channels such as WhatsApp, Telegram, Slack, Facebook Messenger, and Live Chat.\\nThe article assumes that you have a basic understanding of a chatbot application, such as intents, entities, and knowledge corpus training. To understand the impact better, you should have basic development skills in API integration, Python, and the Watson Assistant SDK, as well as fundamental AI knowledge. The article covers:Business context and the use of business FAQs\\nChallenges in building a chatbot from business FAQs when the FAQs are being continuously revised\\nSolution details on how FAQs provided by customers across industry domains can be automatically fed into various components of the Watson Assistant Service\\nIntegration with input channels such as Telegram and WhatsApp\\nA sample use case with step-by-step implementation for building an automated tool that converts a business's FAQ to a ready chatbot solutionWebsite FAQs and chatbots in business\\nFAQs are important for any company because they provide clarity on the company, their products, and their services. Most companies prefer to add an FAQ page on their website, and the key benefits of having the FAQs on the website are:Improving customer experiences by providing answers to standard questions.\\nExpanding the companys online visibility through a search engine . This in turn also helps Google to understand the business better.\\nProviding customers with knowledgeable, trustworthy, and useful information on the business's services or products.\\nAddressing common questions and answers gives confidence to the customer to take the next step of purchasing a product or service.Extending the FAQs and implementing a chatbot solution provides several benefits for a company or enterprise.A single window to reach the customer. It gives customers a more personal experience than a conversation by mail or phone.\\nSupport is available 24x7 as compared to traditional customer service.\\nA company can react faster to customer inquiries, which can increase and drive a companys sales. Additionally, it can help a company to increase customer satisfaction.\\nChatbots are industry agnostic, that is, they can be used in a variety of industries.\\nChatbots can give immediate feedback to customers.Challenges and solutions\\nAutomating customer service tasks lets customer service representatives spend more time on value-added activities, leaving the chatbot to answer standard queries. Converting the available FAQs to chatbot-compatible input can help you automate the solution. To do this, the business FAQ must be enhanced to build various entities and utterances. When the entities and utterances are ready, the FAQ is fed into the AI solution. However, when implementing an FAQ chatbot solution, there can be some challenges.A considerable amount of time must be spent updating the FAQs with entities and utterances.\\nContinuously updating the FAQs requires additional effort to update the input data for the AI solution.The manual effort of creating entities and utterances in the solution can be avoided entirely by using the auto generation of utterances and intents. You can use the business's FAQs as input, but you want to be able to provide the right answers even if the question is asked in various ways. This is where the use of AI algorithms come\",\n",
       " 'https://developer.ibm.com/blogs//articles/develop-an-ai-infused-automation-tool-to-convert-business-faqs-to-watson-assistant-ready-chatbot-input\\n')"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elastic_reranker(question,5,reranker)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b90d664e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
